Resource Management for Scientific Application in Hybrid Cloud Computing Environments

Cloud computing is an emerging commercial infrastructure paradigm that promises to eliminate the need for maintaining expensive computing hardware. Nevertheless, the potential of using Cloud computing infrastructure to support computational and data-intensive scientific applications has not yet been sufficiently addressed. This thesis closes this gap by researching an architecture and techniques for performance and cost-efficient execution of scientific applications on Cloud computing infrastructures, organized in five chapters. First, we investigate the suitability of the workflow paradigm for programming scientific applications, following from their success on related distributed computing infrastructures such as computational Grids. We present case studies for modeling two applications from the astrophysics field as scientific workflow applications to be run with improved performance on multiple leased Cloud resources. We further analyze the workflow traces collected over the last three years of research in the Austrian Grid and classify them according to their different structural and performance characteristics for later evaluation purposes. Second, we investigate the problem of provisioning and management of Cloud resources to large-scale scientific workflows that do not benefit from sufficient free Grid resources, as required by their computational requirements. For this purpose, we propose an extended architecture comprising new services that allow using Cloud resources in an integrated manner with minimal application interface change: resource management for virtualized hardware, software catalogues for machine images, and integrated security and authentication features. The evaluation of the proposed architecture indicates that using Cloud resources for scientific applications is a viable choice, and execution times can be significantly reduced by acquiring additional ondemand Cloud resources. Third, there is currently a lack of models to understand the performance offered by existing Cloud computing infrastructures required for scheduling and running scientific workflow applications. We perform an empirical evaluation of the performance of four commercial Cloud computing services including Amazon EC2, using different benchmarks for single resource instances and virtual clusters. We compare the performance characteristics and cost models of Clouds to other scientific computing platforms such as production parallel computers and computational Grids. The results prove that certain resource types offered by Cloud providers have high potential for speeding up the execution of loosely coupled parallel applications such as scientific workflows, especially for short deadlines. Fourth, to address the lack of scalable simulators to support the Cloud computing research, we developed GroudSim, a Grid and Cloud simulation toolkit for scientific computing based on a scalable simulation-independent discrete-event engine. GroudSim provides a comprehensive set of features for complex simulation scenarios from simple job executions on leased computing resources to file transfers, calculation of costs, and background load on resources. We illustrate real scenarios of using this simulation toolkit to accelerate the evaluation of various optimized resource provisioning techniques by a factor of 700 compared to real execution with no resource cost expenses. Finally, we address the problem of dynamic provisioning of Cloud resources to large-scale scientific workflows with respect to four important aspects: (1) when to extend the Grid infrastructure with Cloud resources, (2) the amount of Cloud resources to be provisioned, (3) when to move computation from Cloud to the Grid and (4) when to release Cloud resources if no longer necessary. Then, we address the NP-complete problem of scheduling scientific workflows on heterogeneous Cloud resources by proposing an extension to the dynamic critical path scheduling algorithm for dealing with the general resource leasing model encountered in today’s commercial Clouds. We analyze the availability of the cheaper and unreliable Spot instances and study their potential to complement the unavailability of Grid resources for large workflow executions. Experimental results demonstrate that Spot instances represent a 60% cheaper but equally reliable alternative to Standard instances provided that a correct user bet is made.